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[57] ABSTRACT 

A real-time three-dimensional display device particu- 
larly suited for medical imaging is disclosed. The device 
includes a plurality of individual processing elements 
each having an image memory for storing a mini-image 
of a portion of the object as viewed from any given 
direction and a merge control means for generating a 
combmed hnage of the object including the depth 
thereof by selection on a pizel-by-pixel basis from each 
of the nuni-images. In two different embodiments, pri- 
ority codes are assigned to each of the processing ele- 
ments reflecting the relative significance of a given pixel 
of the mini-image produced by a given processing ele- 
ment as compared to the pixels of mini-images produced 
by other processing elements. In one embodiment, the 
combined image is generated in accordance with the 
priority codes. In another embodiment, a Z buffer is 
used to provide for hidden surface removal on a pixel- 
by-pixel basis. Improved shadow, shading and gradient 
processors are provided to provide three-dimensional 
imaging as well as an improved scan conversion means 
for generating a coherent image from the combined 
images merged from all of the processing elements. 
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photon emission computed tomography (SPECT) or 
THREE DIMENSIONAL MEDICAL IMAGE position emissfon tomography (PET) is employed, both 
DISPLAY SYSTEM of which rely on digital techniques. In both these mo- 

dalitiesy the information associated with each voxel is a 
BACKGROUND OF THE INVENTION 5 measure of the concentration of radiopharmaceutical at 

The present invention relates, in general, to an im- the corresponding point within the patient SPECT and 
proved medical image display device and, in particular, PET differ from CT and MRI in that the images are 
relates to an improved device for displaying three-di- fundamentally physiological rather than anatomical 
mensional medical data. (although certain MRI studies have a physiological 

Physicians have an important need for knowledge component). Thus, for example, nuclear medicine stud- 
and infonnation of the structure and condition of a ies are used to demonstrate abnormal growth activity in 
patient's internal anatomy. More importantly, physi- bones which otherwise appear normal, 
cians have a need for such information which is ob- Another common imaging modality which yields 
tained by non-mvasive techniques. The need for knowl- three-dimensional information in digital format is diag- 
edge and information of the structure and condition of 15 ^^j^j^ ultrasound. This technique reUes on the reflection 
the mternal anatomy was first fiUed vnth the use of of sound waves at interfaces within the body (e.g., be- 
X-rays. In recent years, however, the X-ray machine ^^een fluid and soft tissue) to generate "echoes"; the 
hi« been supplemented, if not ^tirely rq)Iaced. by elapsed time between the tranan^on of a pulsed siund 
medical unagmg techniques which yield three-dimen- «f 5*« -^^-Tl -!r-l^-^r 

sional infor^Sn. Th^ techniques include, for exam- 20 ^''^.f^l^! 'fjf^^^^^ f.^^ * J 
pie. computed tomography, manietic resource imag. ^^^i^^afcc to the mterface. Transmission ultrasound 
LgandStrasonicsSig: sj^ have been proposed, but these are currently in 

Perhaps the best known technique of this type is the atperunentol stage. C^^^^ 
computed tomography (CT). With computed tomogra- ?P shce-by-shce m a manner analogous to CT 

phy. a complete three-dimensional examination is made 25 {except that the slices are usually longitudinal rather 
up of a sequence of two-dimensional cross-sections or *^ transverse); digital techniques are not needed to 
"slices". Slice infonnation is acquired by rotating a thin, I«oduce the images although almost all modem devices 
fan-shaped beam of X-rays about the long axis of the *tore the image in digital format to avoid electronic 
patient Each slice is irradiated by its edges; the trans- ^ to facilitate post-processing. In this modality, 

mitted X-ray beams are captured with position-sensitive 30 the information associated with each voxel represents 
radiation detectors and thus X-ray attenuation measure- ^ strength of the echo at the corresponding point 
ments are obtained from many different directions within the body; this in turn is a measure of acoustic 
across the slice. The two-dhnensional images are "re- impedance, a flmction of the type of materials present at 
constructed" from these measurements by a mathemati- the mterfoce. 

cal procedure known as "convolution and back-projec- 35 A major drawback of each of the imaging devices 
tion." The output of the reconstructron procedure is an described above is that images are produced which 
array of numbers (known as pixels in 2D, or voxels in comprise two-dhnensional slices of the internal anatom- 
3D) representing the radiological density pc-rayattenu- ical structures being observed. Physicians must then 
atlon) at points within the sUce. mentally "stack" an entire series of these two-dimen- 

A recoiUy developed unaging modality which does 40 sional slices in order to infer the structure of the three- 
JSufp^^S^ magnetic resonance unaging dimensional objects under investigation. Many prob- 
(MRI). This technique uses large magnetic fields to fems are inherent in such an wproach. 
excite protons withm the body ^^JlL^%P^^^om^^ First, the interpretation of aa«ies of stacked, two^- 
non of nuclear magnetic resonance (N^). The excited „ensionaI ima^S^by a physician requiresT^t deal of 
protons produce a radio-frequency (RF) si^ which 45 knowledge^^ld skiU S^nX^^ach ^ 

can be position-encoded. Three-dimensional informa- u • i ^ Tli.. ]t 

tion can be buUt up slice-by-slice. as with X-ray C^r ^^P'^^? ^ ^"^^.^ ^ consummg. Thirdly, the 
an entire 3D voile can be imaged directly. In either ^^Kl" K "^^^T^' ^. , ^. , , . 
case. 2D sUces are reconstructed for viewing, using f^^^ ""^.^ ^ ?^ "^^^^ 

mathematical procedures analogous to those used in 50 produces a three-dunensional representation of 

X-ray CT. In MRI, the infonnation associated with anatomical structures produced from a ftill 

each voxel is a composite of proton density (essentially, °' stacked two-dimensional slices of that struc 

the amount of water present)3nd spncaUed Tl and T2 ^ needed even more is a medical image 

relaxation times, which ar^-fimctions of-physical and ^^P^y ^^^^^ ^^^^'^^ provides the physician or other 
chemical structure. C / 55 ^^bserver with the ability t manipulate the object and its 

Other three-dimensional imaging techniques fall image interactively in real tune such that the object may 
within the reahn of nuclear medicine. The basic princi- viewed from various directions and hi various modes 
pie here is to introduce radioactive substances (radio- ^ real time. By real-time display is meant that the video 
pharmaceuticals) mto the body, relying on their phar- display output should be updated at or near video rates 
macological properties for uptake into spediic organs 60 ^f frames per second. Provided there is mmunal or 
(for example, radioactive iodine can be used to label the delay between operator action and the correspond- 
thyroid). These radioactive substances produce radia- mg change in the final ixnage.. this update rate would . 
tion which may be measured with position-sensitive provide instantaneous perceptual feedback. It should be 
detectors external to the body, known as '*ganuna cam- clear that such an hiteractiveb three-dimensional display 
eras." Two-dimensional projected images (comparable 65 system permitting a physician to visualize and mteract 
to those obtained with conventional x-rays) can be gen- with a shaded, three-dunensional representation of an 
crated with analog electronic circuitry; To obtab reli- anatomical structure would greatly facilitate the exami- 
able three-dimensional mformation, however, single nation of that strucfire m conjunction with medical 



04/16/2004, EAST Version: 1.4.1 



4,737,921 

3 4 

research, clinical diagnoses and the planning of surgical technical difficulties have so far prevented these dis- 

procedures. plays from becoming practical. 

A number of 3D display systems for medical objects Probably the most familiar method of generating 

have been described in the literature, but none of these realistic images from a 3D scene is to project it onto a 

provide realistic shaded images at the full resolution of 5 2D viewing screen, and rely on motion parallax, projec* 

the input data with real-time interactive capabilities. tive geometry, shading and hidden surface removal to 

Three-dimensional medical datasets can be displayed create the iUusion of depth. The result is similar to con- 

m the following ways: CO true 3D images can be created ventional TV and motion pictures, which most people 

in space, or (iO 2 JD images can be generated by project- rciEulfly interpret as representing 3D scenes. In fact, this 

ing objects or object surfaces onto a 2D viewing screen 10 technique can be employed to generate true 3D images 

with depth cues, or (iii) the data can be reorganized into by delivering separate stereoscopic views to each eye, 

asequenceofreprojectcd views or slices. foj example using lightweight viewing glasses with 

Most CT and MR! display systems provide facilities electro-optic shutters which opcrat synchronously with 

to work through a sequence of 2D slices fairly rapidly, |||^ display. 

so that the traned physidancan create a mental impr^ 15 objects represented by ID primitives (contours) can 

swn of the 3D structure. On the other hand, only the ^c displayed in real time on calligraphic equipment 

ongmd slices captured ^y^^i^ from several manufacturers. Typical systems 

mp^ displayed: rc-dicing or re-formattmg the data to ^an display about 100,000 short vectors without appie- 

obtamii^w cro^ ^i^^lc flicker, accommodating, for example. 100 con- 

tional effort Re-shcmg or re-fonnattrng die date to 20 Jinarginally sufTicicnt for 

generatem!^2Dsh««(wito^^ complex medical objects. Contour-based "wireframe" 

I referred to as mdti-planar reco«^^ (MPR . , J ^^^^ ^ ^ depth-cues than their 

SCAN DATA, INVESTIGATIVE RADIOLOGY. 25 '^^"^ capabilities, calUgraphic equipment has 
Vol 10, No. 5, SEPT.-OCT. 1975, and also by Herman ^""f f 1 ? '"^^""^ ^ "^^ a u . 
and Liu, Display of Three Dimensional Informaton in ^ solWe-based syston running on dedicated hard- 
Computed Tomography, Computer Assisted Tomogra- ware which denves shaded surface disptoys from ID 
. phy, Vol. 1, pp. 155-160, 1977. MPR facilities exist on ^ ^^"^^^^ commaically from Contour 
I most CT and MRI display systems, requiring roughly 1 30 System*. Mountain View, Qdifonua. That 
j minute of computer time to generate each new slice system is designed to facditate surgical and radiation 
(dependmg on its orientation). therapy planning, allowing interaction with and modifi- 
True 3D images can be created in space using the contours on a sUce-by-slice basis. The user 
varifocal mirror method described in U.S. Pat. No. has the option of creating ia shaded graphics (2}D) im- 
4,130,832 Sher. In this method, successive slices are 35 ag^ » ™odel can be fobricated on a milling ma- 
displayed on a CRT under computer control. The sUces computer control (See U.S. Pat No. 
are viewed by reflection in a mirror whose vibrations 4,436,684.) Following interaction witii and modification 
arc synchronized with the update of the CRT. As the ^ object, a new surface must be formed for shaded 
mirror surface moves back and forth, different images graphics display. While surface dispUy can be accom- 
(representing different depths) are seen, so that a true 40 P^»hed rapidly (typically, 45 seconds per view), it can- 
;3D image is perceived. Tlie varifocal mirror is a thin ^ ^ 

membrane whose focal length changes when vibrated ^ software-based dedicated physician's workstetion 

so that the virtual images vary over distances of 10-30 which apparently generates 2iD images from voxel 

centimeters. One disadvantege of the varifocal mirror faces was recently introduced by Dimensional Mcdi- 

method is the mechanical nature of this technology. 45 The software generates and stores a number 

The use of vibrating mirrors is less reliable and conve- views of a rotating object and plays them back in 

nient than simply viewing 2D or 2}D images on a TV "movie" mode. A typical movie takes about 1 hour to 

monitor. Another disadvantage is the bandwidth of the generate and offers little or no interactive capabilities at 

CRT itself, since every point in 3D-space must be deliv* display time. 

ered to the screen in the course of each cycle. 30 A number of high-performance systems which are 
Another method of creating a true 3D image is de* based on polygon display and which can generate im- 
scribed in U.S. Pat. No. 4,160,973. This method dis- ages rapidly are available commerically. These can be 
penses with mirrors and CRTs: a planar array of LED's used for medical objects provided the voxels are con- 
is rotated at high speed, and the individual light sources verted to polygons through software preprocessing, 
^e turned on and off at the appropriate times. One 55 The large number of voxels needed to represent com- 
advantage is that dynamically changing objects can be plex anatomical objected however, places severe de- 
displayed smce the LED's can be switched quite xap- mands on throughput Flight simulators represent the 
idly. state of the art hi polygon-based display, both in 
Three-dhnensional displays based on the flourescence throughput (several thousand polygons at 30 frames/- 
ofa gas excited by a laser have been under investigation 60 second) and m hnage quality, but are insufficient to 
for a number of years. In principle, a suitable gas (such display the number of voxels found with complex medi- 
as I*CI vapor) can be made to flouresce at a point in cal objecta. 

space which is at the intersection of two laser beams, Another method of displaying objects built of poly* 

neither of which alone is.suflicientiy intense to exdte gons and having lower performance than flight Simula* 

flourescence. The intersection point can be scanned 65 tor systems utilizes "smart frame-bufTers" designed as 

throughout the display volume and a 3D image built up peripherals of general purpose computers. See, for ex- 

in a manner analogous to the (two-dimensional) scan- ample U.S. Pat No. 4,475,104 Sher. These typically 

ning electron beam in a conventional TV. However, have tiiroughput slower tiian that of flight simulators 
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and are too slow for the real-time display of anatomical mensional final image. The overall display processor 
structures. architecture is based on a principle which maps 3D 

An octree-based 3D di^lay system is available com- object space into 2D image space using either a Z-buffer 
merically from Phoenix Data Systems, Albany, N.Y. or equivalent time ordered display procedure for hid- 
This is a single processor hnplementation of the algo- S den or obscured surface removal. In the latter version, 
rithffls developed by Meagher, and is capable of only for a given orientation of the object, pixels are written 
processing the equivalent of up to 10^ voxels per second mto a 2D image (display) buffer in time order sequence 
(i.e., typical views of complex objects are updated once corresponding to reading out voxels from the back to 
per second) which is too sdow to display the typical 3D the front of the object This approach guarantees that 
medical data set in real-time. 10 any point that should be obscured in the final image by 

It will be seen that none of the existing systems de- something in front of it will, in fact, be invisible in the 
scribed above provides sufficient throughput for real- final reconstructed image. The same result can be 
time display of medical objects with full resolution. achieved by using a Z-buffer which controls a selective 

General concepts of an interactive, real-time, three- replacement strategy, 
dimensional medical object display device which meets IS The imaging device described in the aforementioned 
this need have been set forth in a paper entitled "An pi|blication has a number of limitations and drawbacks 
Architecture for the Real Time Display and Manipula- which have been cured by the present inventors. In 
tion of Three-Dimensionai Objects'* by S. M. Gold- addition, certain important enhancements have been 
wa&ser and R. A. Reynolds, two of the co-inventors of added. These drawbacks and enhancements are de- 
the present application, which was presented to the 20 scribed below: 
1983 proceedh^gs of the International Conference on 

Parallel Processing held in Bel Aire, Mich, in August of ^^8® Scheme 

1983. In that paper, a special purpose multi-processor As described in the foregoing paper, the display corn- 
architecture is described which permits the high speed putation may be distributed among multiple processing 
display and manipulation of shaded, three-dimensional 25 elements. The paper suggests that unless the computa- 
objects or object surfaces on a conventional raster scan tion is distributed among multi-processors, real-time 
CRT. display and manipulation would be impossible. The 

In the foregoing paper, the input to the system is a set number of voxels any particular processor can render 
of digital data representing the three-dimensional, ana- per frame is constrained by memory speed and the time 
tomical object under consideration. This set of data is 30 required to complete the coordinate transform from 
stored in memory which is referred to as "object object space coordinates to image space coordinates, 
space." The problem to which the paper is directed is Because medical data can typically range in the tens of 
that of displaying the set of three-dimensional data m millions of voxels and because a single processor can 
object space on a two-dimensional screen in a manner only render several hundreds of thousands per frame, it 
which retains the three-dimensional nature of the object 35 is desirable to distribute the computation among several 
under consideration. When displaying the three-dimen- processors, each assigned to a local reg;ion of object 
sional data set on a two-dimensional screen, most of the space. If the computation is distributed in this manner, it 
stored data is irrelevant and must be ignored. In the is then necessary to "merge" the mini-images computed 
three-dimensional data set are many data points repre- by the several processors into one final image represent- 
senting portions of the back of the object under consid- 40 ing the entire object space. 

eration relative to the viewer which are obscured to the In the architecture described hi he foregoing paper, 
viewer m the fmal image. Only those data points which the object space is partitioned into 64 subcubes. Each of 
arc in the front of the object relative to the viewer, i.e., these subcubes is comprised of 64X 64x 64 voxels, i.e., 
not obscured, are displayed. In short, the problem to approximately 256,000 voxels, each voxel containing, a 
which the foregoing paper is directed is the problem of 45 single eight-bit tissue density byte. The paper describes 
mapping three-dimensional object space onto two di- how the mini-image from sixty-four processing ele- 
mensional image space while preservmg the three-di- ments can be combined to handle a combined object 
mensional quality of the fmal image and doing so in real space of 256 X 256 X 256 voxels. Each of the processing 

elements computes an individual local, mini-image from 

In the display device disclosed in the aforementioned 50 its region of the partitioned object space. Eight interme- 
publication, the object space is a three-dimensional diate processors then create eight sub-images from these 
scene partitioned by three sets of parallel planes into 64 mini-images by serially reading the mini-images into 
cube-shaped volume elements or "voxels." Associated the sub-images in back-to-front order. These eight sub- 
with each voxel is a numeric quantity reflecting the images are then merged in a similar, back-to-front serial 
density of the object as determined by the imaging de- 55 manner to form the final image, 
vice, Le. MRI, CT or other data. The density, for in- The scheme of merging mini-fanages hito sub-images, 
stance, is an 8-bit byte representing 256 different denaty and sub-images into a final image is cumbersome in 
levels. terms of hardware For each serial merge an alternating 

While the stored voxels of the present invention are double frame buffer is required. This is a total of mne 
typically cube-shaped volume elements, the present 60 double buffers (eight for the first level merges, and one 
invention is not so limited. As will be described in con- for the second level merge). Each double buffer re- 
nection with FIG. 2 below, some medical imaging sys* quires significant memory and boardspace. It would be 
tems such as GT, for example, produce rectangular desirable t simplify the hardware configuration, 
rather than cubic voxels. The present invention is In addition to hardware simplification it would also 
equally applicable to rectangular or cubic voxels. 65 be desirable to reduce the time between operator action 

The aforementioned publication describes multipro- and the time the final image appears on the display, 
cessor hardware and software which retrieves the re- When tiie operator reorients the object or interacts witii 
spective voxel data and displays and generates ? two-di- it in any number of ways, he expects to see an instanta- 
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neous change in the display of the final image. With the ^ . p. 
merge scheme described in the aforementioned publica> ^ ^® 
tion, several video frames elapsed between the time of In the display device of the foregoing paper, a three- 
operator action and the image display of that action. dimensional image is produced. The paper, however, 
This delay is referred to as "pipeline latency." Each 5 nowhere mentions the desirability of also selectively 
merge level results in a delay of one frame. Thus, in the displaying two-dimensional cross sections of stored 
architecture of the foregoing paper three frames elapse three-^limensional objects. Such techniques, however, 
between operator action and a change in the final dis- ^ known to be usefbL In known systems, the original 
play. It would be desirable to improve the merge two-dimensional sUces captured by the imaging system 
scheme of the foregoing p^>cr to reduce the pipeline 10 ^an be presented to the viewer rapidly-in effect flip- 
latency to more acciytable timw. ping through the sUcesofthe object in the direction the 
One additional difficulty found with the aichitectuie ^^^^ ^„ ^ ^^^^ desirable, how- 
of the aforementioa^ paper is that each procemig ^ ^^^^^ physician with the additional capa- 
donent is constramed to procejsmg^ from a givoi of creating arbitrary and new two-dimensional 
subset of^e date iep»a«Ung the object umier consid- 15 > cutting through the object in different planes 
eration. \Wth the architecture described m the afore- ^ ^J^^^^^„ .x,^ Aat^\ tuu ;c «ic/^ 
mentioned publication, object space is partitioned into ^' ^^^ reformattmg the ongmal data^ ^ ^ also known 
STsKS^^uS S^r^xSxS f mult-planar reconstruction (MPR). With such a 
voxebTEa^sub-cube defines a partic^r unvarying f«**"'^ ^ phyacian could better comprehend the 
portion oftheoveranobjectspace.Moreover,theover- 20 "wer structure of the objert mider consideraton It 
all object space is configured in a 256 cube shape ^ould be dttirablc to provide a medical unagmg dis- 
(4X 4X4 sub-cubes). The 64 individual processing ele- P^^^ J^^^^^ ^^^8 » 'yice plane" feature which would 
ments are grouped into eight groups of eight processing provide the mteractive capability of permitting the 
elements. The mini-images from each of the individual physidan to observe any two-dimensional cross-sec- 
processing elements are merged by a group of eight 25 ^onal view of the object taken from any selected direc- 
intermediate processors. The intermediate processors tion. Such a feature would permit the physician to selec- 
are hard-wired to a defined gionp of eight processing lively isolate bodily structures for closer observation, 
elements. It would also be desirable to provide a medical imag- 
With the architecture of the foregoing paper, the iug system having the capability of displaying shaded 

Jobject space is an unvarying cube configuration. There- 30 three-dimensional objects and having the further capa- 

fore, if the object under consideration is an elongated, bility of slicing those objects to view interior structures 

: anatomical structure such as a bone» final image resolu- normally hidden from 20 view. It would be further 

tion is less than optimum because object space is not desirable to provide this reslicing capability from any 

used efficiently. It would be desirable, in addition to arbitrary direction of view. One system with this capa- 

. improving the merge scheme set forth in the previous 35 bility is known which is made and sold by Phoenix Data 

paper, to provide reconfigurable object space to accom- Systems, Inc. This system, however, takes a second or 

- modate anatomical structure of diifering shape. so to display a new sectional view after user interaction. 

Multiple Independently Movable Objects ^uch time periods unduly long to permit effective 

user mteraction. Only real time response permits hand- 
The utiUty of many medical images would be greaUy 40 eye coordination of the user to accurately position the 
^:cnhanced if the display devkre set forth in the foregoing 5^^^ p]g^^ y^f^ respect to the object 
paper had the further ability of independently manipu- 
lating various objects or portions of objects displayed Scan-Conversion 
on the screen. For example, it would be particularly Qne problem associated with the system described in 
desirable to display individual bones of the human skele- 45 aforementioned publication is that of unwanted 
ton or a portion thereof including appropriate linkages u^oles" in the final dispky. As set forth in the paper, it 
at vanous joints It wodd be desirable to provide a ^ ^ ^oles appear in the output image at 
medical miagmg display device m which the physi«an orientations. TTiesc holes occur if object space 
b ter^^io^^"^ "^^^ 50 <Iataismagnifiedasitisdisplayedonafinaltwo-diien- 
The aforementioned paper describes the desirabUity ij?'^ f^^^^ a scale-factor greater than 0.577 (1). 
of providing for the dis^y of independently movable ^^^'^l scan-conversion is 
objects but does not d«cribe or contemplate a tech- P^^orm^^ projected voxels have the appearance of 
Diquc for displaying a multiplicity of objects which may Pomts rather than cubes. At certam orientations and 
be translated and/or rotated with respect to one another 55 niagmfications, there are insufficient pomts to give the 
in the final image. appearance of a solid object on the video output display 

and holes will result. 

Dynamic Display The foregoing paper indicates that certain scan-con- 

It would also be desirable to provide a three-dimen- version or hole correction techniques had been at- 

sional display of time varying anatomical structures. 60 tempted. The first ofthese techniques was the display of 

Specifically, it would be desirable to provide a three-di- the centers of the visible faces of each voxel and the 

mensional display of the heart as it moved from a first second technique was referred to as **double resolution 

contracted or diastolic position through its various pha- interpolation and resampling.'* It has been found that 

ses to a fully expanded or systolic position. It would also neither of the foregoing techniques has solved the prob- 

be desirable to compare objects before and aAer treat- 65 lem. It would be desirable to provide a medical imaging 

ment or to study clumges in the object with time. An- display device which eliminates the poor quality image 

other application is the study of moving joints to study caused by holes in the two-dimensional display of solid 

their mechanics. objects. 
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Shading 

Another difficulty with the medical imaging system 
described in the foregoing publicatioa is that of shading 
the final image. The realistic appearance of computer 5 
generated images is greatly enhanced by controlling the 
bri^tness of each pixel in accordance with the illumi- 
nation the corresponding point of the object would 
receive from one or more simulated light sources. The 
illumination at a given point depends upon the distance 10 
of the object from the simulated light source and the 
orientation of the surface of the object with respect to 
light rays from that simulated light source. 

Shading methods in computer generated images are 
well known in the case where objects are modeled by 
polygons or curved patches. In these approaches, the 
surface orientation of the object being displayed can be 
represented by vertex normals stored with each surface 
element, polygon or patch. However, these prior art 
shading methods cannot be applied to the real-time^^ 
display of objects composed of voxels. The reason is 
that the voxel approach provides facilities to modify or 
change the orientation of the object at the time of dis- 
play and, therefore^ surface shape and orientation (and 
therefore vertex normals) cannot be predicted before 
display. 

in the aforementioned paper, the requirements for 
realistic shading were set forth. In a second and subse- 
quent paper entitled ''Image Space Shading of Three- ^ 
Dimensional Objects" published by one of the co-mven- 
tors of the present application. Technical Report MIPG 
8S, Department of Radiology, University of Pennsylvap 
nia, November, 1983, a mathematical solution to this 
shading problem was proposed which is applicable to 35 
voxel-based object space. Although the possibility of 
performing gradient shading on high-speed hardware 
was discussed, no practical method of real-time imple- 
mentation was given. It would be desirable to provide 
real-time implementation of nnage space shading to 40 
obtain an enhanced a final three-dimensional display in 
a medical imaging display device of the type described 
in the aforementioned paper. 

Shadows 

45 

In addition to reahstic shading effects, three-dimen- 
sional perception in the final display can be greatly 
enhanced by adding shadows to parts of the object 
being displayed which are obscured from the simulated 
light source by other parts of the object. 50 

The generation of shadows in computer images re- 
quires two stages. First, those parts of the object surface 
bdng displayed which are illuminated by the simulated 
light source and those which are not illuminated from a 
simulated light source must be determined. Secondly, 55 
the parts of the object which are obscured from the 
light source but visible to the observer must have the 
brightness adjusted m those locations where shadows 
appear. Methods of generating shadows in computer 
generated images are known to exist However, real 60 
time generation of shadows cast by moving objects has 
either been unavailable or, if available, has only been 
accomplished with extraordinarily expensive hardware 
and with limited performance. See, Schachter, B. J., ed., 
"computer Image Generation", Wiley-InterScience, 65 
N.Y., 1983. It would be desirable to provide a simplified 
shadow generating capability in the three-dimensional 
display of medical images of the aforementioned paper. 



Object Editing 

An important aspect of three-dimensional imaging in 
medical applications is the ability to remove unwanted 
objects, or separate one object from another, in a given 
scene. For example, it would be particolariy desirable 
for an orthopedic surgeon to disassemble a ball and 
socket joint display so that both parts could be exam- 
ined separately. Such segmentation of a given scene can 
be performed interactively by providing an "object 
editing" capability enabluig the editing of the object 
space data base and the removal or deletion of parts of 
objects. The desirability of such a feature was reported 
in the aforementioned paper, but a mean for achieving it 
has heretofore not been obtained. 

Automatic Volume Determination 

Still another feature which would be desirable in the 
medical imaging device described in the aforemen- 
tioned publication would be an imaging device having 
an additional means for quickly and accurately deter- 
mining the volume of selected organs, tumors, bones, 
etc. Such a capability is of particular importance in 
performing comparative studies of tumor growth, treat- 
ment effectiveness, rate of post-operative healing, and 
surgical planning. It would be desirable to provide a 
medical imaging device providing an automatic read- 
out of the volume of the three-dimensional object under 
consideration. 

Interactive Threshold 

Still another enhancement which would be desirable 
in the medical imagmg device described in the afore- 
mentioned publication is the ability to quickly select 
various types of tissue for display with other and differ- 
ent types of tissue being disregarded. For example, it 
would be particularly desirable to provide a medical 
imaging device having the capabiUty of displaying a 
bone and muscle grouping and to interactively display 
only the bone or only the muscle from that grouping 
with the corresponding element bemg removed. 

SUMMARY OF THE INVENTION 

These and other objects are achieved in accordance 
with one embodiement of the present inveniton in 
which a plurality of individual processing elements are 
provided each having an image memory for storing a 
mini-image generated by that processing element. A 
merge control means is provided for generating a com- 
bined image from each of the mini-images by selection 
on a pixel-by-pixel basis. The combined image includes 
the depth of each visible surface of the object as viewed 
from any given direction. 

In accordance with one embodiment of the present 
invention, means are provided for generating priority 
codes reflecting the relative significance of a given pixel 
of the mini-image produced by a given processing de- 
ment as compared to the pixels of mini-images produced 
by other processing elements for a given pixel in the 
combined image produced by all of the processing ele- 
ments and a merge control means is provided which is 
responsive to the priority codes for generating a com- 
bined hnage of the obj^t by selection on a pixel-by- 
pixel basis from the mini-images. The foregoing ar- 
rangement includes much ampler hardward than that 
set forth m the Goldwasser and Reynolds paper de- 
scribed above. It also reduces the pipeline latency of the 
merge scheme set forth therehi. . 
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In accordance with another aspect of the present nuQnjiPTTmM nt? tmp pnFTTcnncn 

invention, logic means are provided for selectively gen- SJboD^OT^^^^ 

crating a low priority code for pixels of the mini-image cxnow 

of any given processing element. Such a feature permits Referring now to FIO. 1, the medical image display 

the inclusion (tf a slice plane feature. S system of the present invention will be seen generally at 

In accordance with another important aspect of the 10. The medical image display system includes a host 

present invention^ a means is provided for sdectively computer 18, an image display processor 20 and a CRT 

mapping points to the image memory of each process- display 22. As shown in FIG. 1 a plurality of medical 

ing element having a predetermined set of densities. unage acquisition devices 12, 14 and 16 are shown cou* 

Such a feature permits object editiqg as described 10 pled to the system of the present mvention. It should be 

above: recognized that additional medical image acquisition 

In accordance with stU other hnportant aslects of the devices could, of course, also be utilized. The medical 
present invention, a volume feature counter is provided unage acquisition devices 12, 14 and 16 may comprise 
for accumulatmg a number of points stored in a given CT scanners, MRI scanners, nuclear medicine scanners, 
mini-image of a given processing dement having a non- 15 ultrasonic scanners or any other medical imaging de- 
zero density dimension for automatic volume determi- vice capable of generating digital data representing a 
nation. three-dimensional anatomical object under mvestiga- 

In accordance with still another important aspect of tion. In the system of the present invention, such data is 

the present invention improved scan conversion means directed to the host computer 18. The host computer 18 

are provided as well as an improved shading processor 20 is responsible for archiving and loading Expropriate 

and shadow processor. three-dimensional image files from each of the medical 

^„ ^« ^ «r^,^« image acquisition devices 12, 14, 16, etc. Each of the 

BRIEF DESCRIPTION OF THE DRAWINGS thr^^nsional image ffles stored in the host com- 

The present invention will be more ftiUy understood puter 18 are directed to the image display processor 20 

by reference to the accompanying drawings, in which: 25 of the present invention which generates a dynamic, 

FIG. 1 is a functional block diagram of the three-di- two-dimensional image of the three-dimensional object 

mensional medical image display system of the present for display on a conventional raster scan CRT 22. The 

invention coupled to a plurality of medical image acqui- image produced by the image display processor 20 may 

sition devices; also be directed to the host computer 18 for archival 

FIG. 2 is a schematic diagram depicting a typical 30 purposes, 

two-dimensional slice of imaging data generated by a Referring now to FIG. 2, the data generated by each 

single medical imaging device; of the medical image acquisition devices 12, 14 and 16 

FIO. 3 is a functional block diagram of the medical will be seen. Each of the medical image acquisition 

image display system of FIG. 1; devices 12, 14 and 16 generates a series of slices, oiie of 

FIG. 4(a) is a functional block diagram of one of the 35 which is shown at 24 from the patient 26. Each slice 

processing dements or PE's shown in FIG. 3; comprises an array of voxels one of which is shown at 

FIG. MP) is a detafled block diagram of the Dynamic 28, each having a predetermined X and Y position. A 

Object Thresholding and Slicing unit of the processing plurality of slices such as one shown at 24 are taken in 

element shown in FIG. 4(a). the Z direction to obtain a three-dimensional data set 

FIG. 4(c) is a detailed block diagram of the Volume 40 representing the anatomical object under consideration. 

Feature Counter of the processing element of FIG. 4(a); As shown in FIG. 2, the voxd 28 is a non-cubic or 

FIG. 5 is a detailed block diagram of the coordinate rectangular voxel in which the dunension m the Z di- 

transform unit of FIG. 3; rection exceeds the dimension in either of the X or Y 

FIG. 6(a) is a detailed block diagram of a first em- directions. As mentioned above, the present invention 

bodiment of the merge control unit of FIG. 3; 45 finds utility in the display of medical images generated 

FIG. 6(b) is a detailed block diagram of a second from both cubic and non-cubic voxels. Also, as men- 
embodiment of the merge control unit of FIG. 3; tioned above, each of the medical image acquisition 

FIG. 6(c) is a detailed block diagram of a third em- devices 12, 14 and 16 produces a series of tissue density 

bodiment of the merge control unit of FIG. 3; bytes (D), each typically, but not necessarily dght bits, 

FIO. 7 is a detailed block diagram of the slice plane 50 assodated with each voxel. This tissue density byte 

controller of FIG. 3; reflects the tissue density at that point of the anatomical 

FIG. 8(a) is a block diagram of the scan conversion structure. In short, each of the medical image acquisi- 

unit of FIG. 3; tion devices 12, 14 and 16 produces a three-dimensional 

FIG. 8(6) is a detailed block diagram of the serpentine array of density values, one for each voxd such as 

buffer of the scan conversion unit of FIG. 8(a); 55 shown at 28 which are directed to the host computer 18 

FIG. 8(c) is a detailed block diagram of a smgle dis- and stored by X6, Yo and Zo position. Gated scanners 

tance comparator dement of the serpentine buffer can gather 4D data sets by sampling in time. The pres- 

shown in FIG, 8(a); ent invention also finds utility in the display of 4D data 

FIG. 9(a) is a block diagram of the shading processor which includes a time dimension, 

shown in FIO. 3; 60 Referring now to FIG. 3, the image display system 10 

FIG. 9(b) is a block diagram of the serpentine buffer of the present invention will be described in detail. As 

portion of the gradient processor of FIG. 9(a); shown m FIG. 3, the image display processor 20 of the 

FIG. 9(c) is a block diagram of the gradient computa- present invention is coupled to the host computer 18 via 

tion unit of the gradient processor of FIG. 9(a); a host bus 30. Voxel data from the host bus 30 is di- 

FIG. 10 is a detailed block diagram of the shadow 65 rected from the host computer 18 to an object access 

processor of FIG. 3; and unit 32 which will be more folly described below. The 

FIG. 11 is a detailed block diagram of the object object access unit 32 is a high speed interface which 

access unit of FIG. 3. permits communications with the host computer system 
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18. The object access unit 32 permits high speed object frame buffer 60 permits scan conversion to standard 

data analysis and editing by the host computer 18. The video format for display on a monochrome or color 

object access unit 32 is responsive to the timing and raster scan TV monitor 22. The frame buffer 60 is di- 

control unit 62 and receives timing and control parame- rectly accessible by the host computer 18 and is avail- 

tets from it. Voxel data from the object access unit 32 b 5 able to display other pictorial data or text In addition, 

directed to the object access bus 34v to a plurality of any displayed image may be read back to the host 18 for 

individual processing elements or PE's, four of which archiving or further processing, 

are shown at 36r 38, 40 and 42. In the preferred embodi- A master tuning and control unit 62 is provided 

ment of the present invention, 64 individual processing which is coupled by a control bus 64 to each of the 

elements are provided As will be more 20 fully ex- 10 processmg elements 36, 38, 40 and 42 through the coor- 

plained below, each of the processing dements such as dinate transform unit 48 and the slice plane controller 

36, 38, 40 and 42 include an associated object memory 50, to the merge control unit 46, to the scan conversion 
for storing a first digital representation of a portion of unit 54, to the shading processor 56, to the shadow 

the three-dimensional object to be displayed In other processor 58 and to the high resolution frame buffer 60. 

words, each object memory stores a predetermined and 15 The control bus 64 also serves to communicate data 

variable subset of voxel data transmitted from the object between the host bus 30 and the frame buffer 60. 

access unit 32. Each of the processing elements also To better understand the description that follows, a 

includes an image memory for storing a second digital definition of certain terms is desirable. As used in tliis 

representation of a two-dimensional mim-image of that application, the following terms have the following 

portion of the object. 20 meanings: 

Each of the processing elements 36, 38, 40 and 42 Original Object Coordinates (Xo, Yo, Zo): Xq, Yq, and 

compute a mini-image from the subset of voxel data Xq define the position of a given voxel of data as it was 
stored in its associated object memory. The outputs of acquired by a given medical image acquisition device 

each of the processing elements 36, 38, 40 and 42, etc., such as 12, 14 and 16 as shown in FIG. 1 and FIO. 2. 

are directed to a merge bus 44 and ftom the merge bus 25 object Coordinate or PE Object Coordmatc (X, Y, 

44 to a merge control unit 46. Z): X, Y, Z define the position of a given voxel as 

Each of the processmg demente 36, 38, 40, 42 etc. ^ stored in the object memory of a given processing 

operate under the control of a coordinate transform umt element such as 42. 

48 and a slice plane controller 50 which will be de- Mini-image Coordinate or PE Image Coordinate (X'. 
scribed below. The coordinate transform unit 48 is re- 30 y, and ZO: X', Y'. Z' define the position of any given 
sponsible for computing the image space coordinates y^^^^ i^j™ memory of a given processing de- 
fer the mini-image produced by each processing de- j^^^ 43, x', Y' and 2', therefore, define any 
ment based upon the position of the corresponding p^i^t in the mini-unage produced by a given processing 
mput voxel. The operation of the coordmate transform element 

unit 48 uj based upon a recursive decomp(»ition of the 35 combined Image Coordinates (X", Y", and Z"): X", 

nme ordered back-to-frontdjsplay algorithm used for y", Z" define the position of a given voxd in the com- 
hidden surface removal. The overaU computation of ^ined image produced by merge control unit 46. The 

unage space coordmates is accomplished recursively combined image coordinates (X", V, Z") are rdated to 

starting With the largest sub-cubes and working down ^^e PE image coordmates (X', Y', Z*) by the foUowing 

to mdividual voxds, dividmg by two at each step. 40 reJation^M- 

The merge control unit 46 produces a combined ™' 

image from the individual mini-images produced by jr'»r+jr-of&et 
each of the processing dements 36, 38, 40, 42, etc. In the 

preferred embodiment of the present invention, this . TT^r+Y-ofket 

combined image is directed to a video post- processing 45 

unit 5Z The video post-processing unit 52 enhances the 2" =r +z-ofrset 
combined image and generates a two-dimensional 

shaded image from it by diminating the presence of where X-offset, Y-o^set and Z-offset define the relative 

holes in the final display, by adding realistic three-di- position of the mini-image produced by a given process- 

mensional shading and by adding shadows where ap- 50 ing element with respect to the combined image, 

propriate. The elimination of holes or diasing in the With these definitions in mind, the various parts of 

final image is accomplished by a scan conversion unit 54 the image display processor 20 of FIO. 3 will be ex- 
to which the fbial image is first directed. The output of plained. 

the scan conversion unit 54 is directed to a shading n * ci « 

processor 56 which adds appropriate shading. The out- 55 Processing Elements 

put of the shading processor 56 is directed to a shadow Referring now to FIO. 4(a), an individual processing 

processor 58 which adds the requisite shadows. element such as 42 will be described in detail, it being 

Each of the scan conversion unit 54, the shading understood that the remaining processing elements are 
processor unit 56 and the shadow processor unit 58 identical As shown in FIO. 4(a), the individual process- 
operates on data from the merge control unit 46 m rast- (0 ing dement 42 includes an associated object memory 70 
er-scan order in red-time. Thus, while the use of all are to which voxel data firom the object access bus 34 is 
preferred, the present invention finds utility if any or all directed. Each object miemory 70 preferably comprises 
of these units 54, 56 and 58 are employed. a 2S6K, 16 bit memory which stores a first digitd repre- 

The two-dimensiond shaded image firom the post- sentation of a portion of the three-dimenaiond object to 

processmg unit 52 is directed to a high resohition firame 65 be displayed. Eight or sbcteen-bit tissue density bytes, 
buffer 60 where it is stored. The hi^ resolution frame D, representing that portion of the object are stored in 

buffer 60 preferably comprises a 1024 x 1024 double the object memory 70 at appropriate X, Y and Z ad- 
buffer. The frame buffer 60 stores the find image. The dresses. 
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Each individual processing element such as 42 further by X', Y', position whenever the merge control unit 46 
includes an image memory 72 for storing a second digi- is processing a pixel of the combined image (X", Y") to 
tal representation of a two-dimensional mini-image of which the given processing element might contribute its 
the same portion of the object as is stored in the object data* At that time D' and Z' values, stored in the image 
memory 70. This image memory 12 con^rises an inter- 5 memory 72, are transferred to delay buffer 80. At all 
mediate alternating double buffer which stores such other times, the merge readout control unit 82 generates 
mini-image at X' and Y' addresses. X' and Y' represent an address invalid code to the gate 102 of the priority 
the horizontal and vertical coordinates of the miniim- code generator 78. 

age. These second ctigital representations stored in While one type of processing dement is described in 
image memory 72 include the depth Z' of the mini- 10 connection with FIG. 3, it should be clear that a variety 
image from a predetermined plane along with the modi- of P£ types are possible, for instance, those which ren- 
ted density D' of that mini-image, der polygons, text overlay, isodose distribution or 
Each of the individual processing dements such as 42 which generate images in any number of ways. The key 
further includes a means for mapping the first digital is that the individual processing dements render mini- 
representation of a portion of the object stored in the 15 images themsdves, under the direction of the host. This 
object memory 70 into the second digitd representation allows a dramatic speedup in the generation of the com- 
of a two-dimensional mini-image of that portion of the bined image. 

object stored in image memory 72. This mapping means Referring now to HO. 4(b), the dynamic object thre- 
comprises X, Y, Z address bus 71 from the coordinate sholding and slicing unit 76 of each of the respective 
transform unit 48 which is directed to the object mem- 20 processing elements will be described m detail Each 
ory 70 and the X', Y' address bus 75 and Z' data bus 73 dynamic object thresholding and slicing unit 76 in- 
from the coordinate transform unit 48 which is directed eludes a first density look-up table 500. Eight or sixteen- 
to the image memory 72. X, Y and Z addresses define bit density values, D, from the object memory 70 are 
any point in the object memory whereas X' and Y' directed to the density look-up table 500 via line 501. 
addresses define any point in the image memory 72. At 25 These 16 bits typically comprise 12 bits of density data 
appropriate X\ Y* addresses In image memory 72 are as well as a four-bit tag identifier field. This tag identi- 
stored the requisite values of D' and Z'. fier field is used to facilitate interactive surgical proce- 

A write enable means 77 is provided, the image mem- dures in a manner which will be described bdow. The 
ory 72 being responsive thereto. The write enable first 12 bits of density data, D. from the object memory 
means 77 insures diat only non-zero density values D' 30 70 are passed through the density look-up table 500 to 
are written into the image memory 72. Furthermore the produce, for example, an eight-bit mini-image density 
write enable means 77 is coupled to the volume feature value, D', which is directed via line 503 to the feature 
counter 520 which win be described in connection with look-up table 502. Both the density look-up table 500 
FIG. 4(c) bdow. The volume feature counter 520 auto- and the feature look-up table 502 operate under the 
matically computes the volume of the anatomical object 3S control of the dynamic operating threshold and slicing 
under consideration. unit control means 504. The dynamic object threshold 

In accordance with the preferred embodiment of the and slicing control unit 504 further controls a compara- 
present invention, each of the processing elements such tor 506. Density vdues, D, firom the object memory 70 
as 42 preferably include a dynamic object thresholding are directed to the comparator 506 via line 507 and the 
and slicing unit 76 which is coupled between the object 40 output firom the comparator 506 is directed to the fea- 
memory 70 and the image memory 72 and which will be ture look-up table 502 via line 509. The value of the 
described in coimection with FIG. 4{b) below. Each aforementioned tag identifier field is utilized by the 
processing element such as 42 further comprises a dynamic object threshold and slicing control unit 504 to 
means 78 for generating a priority code reflecting the control the comparator 506 such that the density value 
relative significance of the mini-image produced by that 4S D' may be preselected interactively to include only, for 
processing element for each pixel in the combined example, voxels having the density of bone or muscle, 
image produced by the display device. The output of etc., thereby permitting interactive surgical procedures, 
the priority code generating means 78 is directed to the The dynamic object thresholding and slidng control 
merge control unit 46 via merge bus 44. The output of unit 504 passes mini-image density values D' through 
the image memory 72. of each processmg dement such SO the feature look-up table 502 depending upon whether 
as 42 is coupled to a delay buffer 80 to which two-di- the particular voxd being processed is in front of or 
mensiond mini-images firom the image memory 72 are behind a slice plane. This is determined by slice plane 
directed and stored on a pixd-by-pixd basis. The output control signals, such as SPO or SPl (i.e., for two inde- 
of delay buffer 80 is coupled to the merge bus 44 via an pendent slice planes) which are directed to the feature 
enable gate 81 which is under the control of merge 55 look-up table 502 from the slice plane controller 50 
control unit 46. The merge control unit 46 thus gates the wtuch will be described in detail bdow. 
mmi-image from a given processmg dement 42 to In addition, the dynamic object threshold and slicing 
merge bus 44 if, depending upon its priority code, that control unit 504 may be used to generate dynamic im- 
mlni-hnage will contribute to the find hnage. ages. Specifically, the control means 504 may be utilized 

Each processing element 42 also indudes a merge 60 to generate mini-image density vdues which are time 
readout control umt 82 which recdves input from the varying. One technique for displaying time varying 
host 18 and the merge control unit 46 and which directs data, for example, is to partition each 16-bit object mem- 
output to the image memory 72 and the priority code ory word into sub-words, each sub-word rq>resenting 
generator 78. the density vdue of a given voxel at a particular time. 

The merge readout control unit 82 of each processing 65 This data may be collected utilizing the medical image 
element is responsive to the merge control unit 46. The acquisition devices 12, 14 and 16 utilizing gated scan- 
merge readout control unit 82 includes a timing means ning techniques well known in the art. Utilizing the. 
which retrieves image data stored in image memory 72 dynamic object thrediolding and slicing unit 76, the 
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appropriate portions of the 16-bit words, in object mem- 
ory, based on the selected time may be processed and 
displayed to yield a combined image of a given anatomi- 
cal object as it exists at different periods of time. 

Referring now to PIQ. 4(c), the volume feature S 
counter 520 wiU be described in detail. The volume 
fisature counter 520 is used to automatically determine 
the volume of each object under consideration. The 
volume feature counter 520 includes a plurality of pro- 
grammable read only memories or PROMS 522 there 10 
being one such PROM 522 for each eight individual 
processmg dements sudi as processing element 42 
shown m FIG. 4(a). The volume feature counter 520 
further comprises an adder 524 to which the outputs of 
each of the eight PROMS 522 are directed. The output 15 
of the adder 524 is directed to an accumulator 526 
which operates under the control of the timing and 
control unit 62. The output of the accumulator 526 is 
directed to the host 18 via host bus 30. 

As each processing element steps through all of the 20 
voxels an the object memory 70, the density values D' of 
voxels to be displayed are directed to the intermediate 
alternating double buffer 72. These same signals are 
directed to PROMS 522 of the volume feature counter 
520. Thus, for each voxel which might contribute to the 25 
final display* a write signal is transferred to the volume 
feature counter 520. lliese write signals are combined 
at the adder 524 and accumulated at the accumulator 
526. The accumulated voxel count, which provides a 
volume indication, is thus directed to the host 18. By 30 
suitably programming the dynamic object threshhold- 
ing and slicing unit control means 504 described in 
connection with FIG. 4(6) above, the volume feature 
counter 520 may accumulate a number of voxels, each 
having a pre-detennined density of interest or, altenia- 35 
tively, each having a range of pre-determined densities. 

Coordinate Transformation 

Referring now to FIG. 5, the coordinate transform 
unit 48 will be described in detail The details of the 40 
coordinate transform unit are set forth in the aforemen- 
tioned paper of Goldwasser and Reynolds entitled "An 
Architecture for the Real Hme Display and Manipula* 
tion of Three-Dimensional Objects". In the embodi- 
ment described in FIG. 5, a single coordinate transform 45 
unit 48 is shown. The single coordinate transform unit 
48 shown in FIG. 5 provides X, X', Y, Y', Z and Z' 
addresses for one of the 64 processing elements 36-42 
shown in FIG. 1. It is understood that sixty four coordi- 
nate transform units such as 48 are thus provided, one 50 
for each processing element It should be understood 
that a single, centralized coordinate transform unit 48 
could be provided to serve all of the 64 processing 
elements. This approach reduces the hardware require- 
ments. Nevcrthdess, in the preferred embodiment, sep- 55 
arate, distributed coordinate transform units are pre- 
ferred because the display of multiple independently 
moveable objects is facilitated with such an approach. 

It should also be clear that the object address XYZ 
may also have a T (time) or other 4th dimensional com- 60 
ponent to facilitate the display of dynamic data as men- 
tioned above. 

As shown in FIG. 5, the coordhiate transform unit 48 
preferably comprises an 18-bit counter 90 driven by 
clock pulses from the tuning and control unit 62. The 65 
counter 90 drives four registers referred to as SCT 0 and 
SCT X, SCT Y and SCT Z respectively. Register SCT 
0 produces a series of address codes (X, Y and 2^ which 



are directed to the object memories 70 of a particular 
processing element. The registers, SCT X, SCT Y and 
SCT Z ultimately produce a second series of address 
codes (X', Y' and ZT) which are directed to the image 
memory 72 of that processing element. Each of the 
registers SCT 0 and SCT X, SCT Y and SCT Z are 
loaded by the host 18 once for each orientation of the 
object to be displayed. For each such orientation of the 
object, a given pdnt defined by X Y and Z coordmates 
m the object memory 70 will correspond to a point 
defined by X', Y' and Z' coordmates m the image mem* 
ory 72. 

The outputs of each of the registers SCT X, SCT Y 
and SCT Z are added at adders 300, 302 and 304 to 
appropriate offsets to yield X', Y'and Z'. 

Once the coordinates of a given mini-image are trans- 
formed and directed to the image memory 72 for each 
processing element, it is necessary to merge that mini- 
image with mini-images formed at other processing 
elements to form a combined image. An important as- 
pect of the present invention resides in the improved 
technique of merging such mini-images which will be 
described in connection >vith FIGS. 6(a), 6(6) and 6(c) 
below. 

Merge Scheme 

In accordance with the most important aspect of the 
present invention, the merging of mini-images produced 
by each of the processing elements such as 36, 38, 40 
and 42 is conducted under the control of a merge con- 
trol unit 46 which responds to priority codes from each 
of the processing elements such that a final image is 
produced which includes only those portions of the 
object to be displayed, i.e., those which are not ob- 
scured by other portions of the object as the object is 
seen from a predetermined point The unobscured por- 
tions are selected from the obscured portions in accor- 
dance with the priority codes. 

A first embodiment of a merge scheme useful m con- 
nection with the present invention may be seen in FIG. 
6(a). As shown in FIG. 6(a), the processing element 42 
includes the image memory 72, the merge read-out 
control unit 82, the delay buffer 80, and the PE enable 
81 described above. The processing element 42 further 
includes the priority code generation means 78 which 
comprises gates 100 and 102. The mput to gate 100 is 
coupled to the output of the image memory 72 whereas 
the mput to gate 102 is coupled to the output of gate 100 
and also to the merge read-out control unit 82 as shown. 
The priority code generation means 78 generates a first 
priority code which may be either a high or low value. 

The first priority code generated by the priority code 
generation means 78 is either a high or a low value 
depending upon whether the mini-Image of that pro- 
cessing element contributes to the combined image. A 
given processing element is assigned a low priority by 
priority code generation means 78 whenever the image 
memory 72 (and, therefore, the associated object mem- 
ory 70 of that processing element) contains zero values 
as determined by the gate 100. Additionally, a low pri- 
ority code is assigned whenever the image memory 72 
for that processing element contains non-zero values 
which are situated between a slice plane and a predeter- 
mmed pomt at which a viewer is situated as determined 
by the threshold and slicing unit 76. 

In the embodiment shown in FIG. 6(a), processmg 
group logic 104 is provided which is coupled to the 
output of the priority code generation means 78 of four 
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separate processing elements. This processing group 
logic 104 detennines the relative priorities of four of the 
64 possible processing elements. The processing group 
logic 104 includes a table 106 containing priority rules 
determining which of four processing elements contrib- S 
ute to a given pixel of the combined image. It should be 
understood that in an embodiment containing 64 pro- 
cessing elements* 16 processing group logic blocks such 
as processing group logic 104 are provided The sixteen 
processing group logic blocks 104 generate a second 10 
priority code reflecting which of four processing ele- 
ments connected to each might contribute a given pixel 
of the final display. The second priority codes from all 
other processing group logic blocks such as 104 are 
directed to an intergroup logic block 108. The inter- 15 
group logic block 108 includes four Ubles 110, 112, 114 
and 116, each coupled to the outputs of four processing 
group logic blocks such as 104. Each of the four tables 
110, 112, 114 and 116 are ftirther coupled to a fifth table 
.118, the output of which is a third priority code indicat- 20 
ing which processing group such as 104 and which 
processing element in that group such as processing 
element 42 will contribute to a given pixel of the com- 
bined image. Tliis third priority code is directed via line 
120 to the merge control means 46. The merge control 25 
means 46 includes a P£ enable block 124 and a Z-offset 
table 128. The PE enable block identifies and enables 
the selected processing element such as 42 which will 
contribute to the final combined produced at a given 
pixel. The PE enable block 124 is coupled by line 126 to 30 
the PE enable gate 81. Upon receipt of the third priority 
code from intergroup logic 108, an enable signal is di- 
rected from the PE enable block 124 to the PE enable 
gate 81 to transfer image data from the delay buffer 80 
to the merge bus 44. In addition, the third priority code 35 
from intergroup logic 108 is directed via output line 120 
to the Z-offset table 128. 

The Z-offset table is coupled to an adder 129 which is 
coupled between the merge bus 44 and the video post- 
processor 52. The adder 129 computes the absolute 40 
depth, i.e., the Z" coordinate of the combined image 
• where: 

Z"=Z offeet+Z'; and 

Z'=the depth coordinate of the voxel with respect to 
the processing element mini-image; and 45 

Z-offset relative offset of the mini-image with re- 
spect to the combined image. 

Thus, the input to the video post-processor 52 is a 
combined image from all of the various processing ele- 
ments comprising a series of data in raster scan order 50 
including density values D' and depth values Z" for 
each pixel of the final display at CRT 22. 

Referring now to FIG. 6(b), a second embodiment of 
merge scheme useful in connection with the present 
invention may be seen. Like the embodiment of FIO. 55 
6(a), each processing element such as 42 includes an 
image memory 72, a merge read-out control unit 82 and 
a priority code generation means 78 as described above. 
Unlike the embodiment of FIG. 6(a), however, the 
processing elements such as 42 do not include the delay 60 
buffer 80. Instead, a single delay buffer 80 services a 
group of four separate processing elements such as 36, 
38, 40 and 42 taken together. The outputs of each of the 
processing elements 36, 38, 40 and 42 are directed on a 
group bus 130 to the delay buffer 80 and from the delay 65 
buffer 80 to the merge bus 44. 

As in the embodiment of FIG. 6(a), the merge scheme 
of FIO. 6(6) includes processing group logic 104 to 



which the priority code generators 78 of each process- 
ing element of a given group of processing elements are 
directed via lines 105. The processing group logic 104 
includes a priority table 132 which is loaded once for 
each given orientation of the object to be displayed and 
which produces a second priority code or enabling 
signal indicated by arrow 126 which determines which 
processing element of a given group of processing ele- 
ments will contribute its image data to produce a given 
pixel in the combined image. The second priority code 
stored in the priority table 132 thus indicates which of 
the four processing elements connected to it may con- 
tribute to the combined image on a pixel-by-pixel basis. 
That second priority code is directed to intergroup 
logic 108 via line 109 where the second p iority code is 
compared with the second priority code from another 
group of four processing elements. The intergroup logic 
108 produces a third priority code which is directed via 
lines 111 to the merge control unit 46 where it is com- 
pared with similar third priority codes from seven other 
intergroup logic blocks such as 108. The third priority 
code from the intergroup logic block 108 thus identifies 
which processing dement of two groups of four pro- 
cessing elements will contribute its image data to a 
given pixel of the final image. The merge controller 46, 
in turn, determines which processing element in Che 
eight groups of eight processing elements will contrib- 
ute its data to the combined image. 

The merge control unit 46 comprises a compare- 
exchange tree comprising compare-exchange blocks 
138-150. A first level of compare-exchange blocks in- 
clude blocks 138, 140> 142, and 144. The third priority 
code from two intergroup logic blocks 108 are directed 
to that compare-exchange tree. A second level of two 
compare-exchange blocks 146 and 148 are arranged 
such that compare-exchange block 146 is coupled to the 
outputs of first level compare exchange blocks 138 and 
140 whereas the second level compare-exchange block 
148 is coupled to the outputs of compare-exchange 
blocks 142 and 144. A third level compare exchange 
block 150 is coupled to the outputs of second level 
compare exchange blocks 146 and 148. The compare 
exchange tree comprised of compare exchange blocks 
138-150 is coupled to a logic block 152 which deter- 
mines which processing group and, therefore, which 
processing element in that group will contribute data 
for a given pixel of the final image. The output of logic 
block 152 is directed to the delay buffer 80 of the appro- 
priate processing group. Receipt of an enable signal at 
the appropriate delay buffer 80 transfers data from the 
appropriate processing element from the delay buffer 80 
to the merge bus 44. The data transferred to the merge 
bus 44 from the selected processing element includes an 
eight-bit tissue density bytie D and the Z\ or image 
depth values originally stored in the image memory 72. 

The output of compare exchange block 150 is also 
directed to a Z-of&et table 128 the output of which is 
directed to an adder 129. Also directed to the adder is 
the Z' or depth value of the selected processing ele- 
ment. Like the embodunent of FIG. 6(a), the adder 129 
computes the Z" coordinate of the combined image 
such that: 

Z"=Z-offset+Z'. 
As in the embodiment of FIG. 6(a), values of D' and Z' - 
are directed to the video post-processor 52 hi raster scan 
order on a pixel-by-pixel basis. 

Referring now to FIG. 6(c), a third embodiment of a 
merge scheme useful in connection with the present 
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invention will be described. As in the embodiments of ute its image data to a given pixel of the combined 

FIGS. 6(a) and 6(^X each individual processing element image of all of the processing elements may be varied 

such as 42 includes an image memory 72» a merge read- from one configuration of the object to the next The 

out unit 82 and a priority code generating means 78. As generation of second and third priority codes, for exam- 

in the embodiments of FIGS. 6(a) and 6(6)» the priority 5 pie, is table driven. Depending upon the priority rules 

code generating mean 78 includes gates 100 and 102 stored in such tables, different ones of the overaU group 

which generate a first priority code having two states, of processing elements may be caused to contribute to 

i.e.» a high priority and a low priority value. the combined image. Thus, for example, the processing 

In the embodiment of FIG. 6(c), the output of image elements 36, 38, 40, 42, etc. may be reconfigured to 

memory 72 is also directed to an adder 160 where the Z' 10 better represent, for example, a long, thin object as 

address of nnage data is summed with the Z offset value opposed to a round object The priority codes may be 

firom Z offset register 162. The summation of the Z' adjusted to ensure that all of the processing elements are 

address from image memory 72 and Z-oi&et value from utilized together t mMimige resolution in the appropri- 

the Z*offset register 162 is the absohite depth Z" of the ate areas. 

given pixd from a predetennined point in 15 the preferred embodiment of FIG. 6(c), which 
ima^. It shoidd be clear that Z is equ^ to the sum of employs a Z compare priority determination on a pixel 
Z +Z offset Upon receipt of a first high priority code ^y pixel, raster scan basis, the maximum in flexibUity is 
from gate 102^e means for g^cratong a pm^ achieved. For example, with that embodiment, portions 
78 directs the Z values from the adder 160 to process- ^ combined image may be interactively translated 
mg group logic 104. In the embodiment of 6(c), there- 20 ^ ^^^er portions and with respect to the 
fore, the first priority code generated by the prionty combined image. In fact, those portions may be trans- 
code generatmg means 78 reflects the absolute d^ lated so as to overlap each other. Moreover, if each 
of a given pixel produced by tha^ proc«smg element in processmg element is provided with its own associated 
the combmed miage. Z valu^m each group of four coordinate transform init 48, portions of the combined 
piDcessing elemente such^ 36^2 are directed to the 25 ^ independently reoriented with respect to 
processing group logic 104 where the Z vabes a^e To^ompbsh this objective some sub- 
compared m a four.toK>ne coi^are^change tree 164. processing elenients may be assigned to pro- 
The compare-exchange tree 164 selects the processmg j„„^ « ^iZi * «uj^«* 

element 36^2 having the lowest absolute Z" value, U . """-"P^S^ * P^'^'^" ^ ^^^^c* 

» w «*w i^w*«v awavitt*^ *Muj,*^., othcT proccssmg clcments BTc assigncd to produce miui- 
the processmg d,^ent producing a pwel closet to the 30 ^^^^ ^ ^^^^ P ^ ^ 

vantage pomt of the observer of the combmed image. obiwt 

The compare exchange tree 104 thus enables the pro- n r ^ _*_*r 

cessing el^nt 36^2 having image data with the 1^- ^ These fcatuas a« nnportmt. for example, m perm . 

^pZfity to be displayed for a |ven pixel to advaice ^| * Phy^ician to «*unme mov«g jomts such as a baU 

that data to delay buffer 80. The ^c«^ing group logic 35 J""*' 

104 further produces a second priority code also con- Slice Plane Controller 

sisting of Z" value for the selected processing element n r • * ct/^ ^ *i. i * « «a 

36-42 and directs that second priority code to inter- ^^^""^8 7. the shce ptone controller 50 

group priority logic 108 where it is compared with the '^^J^^'^^ ^f^* "r^" 

Z" vdues from iSother group of four processing ele- 40 be desonbed to detjiL The sUce ptane conttol- 

ments. The intergroup lo|c 108 generat^ a third prior- ^ ^ P«!!**" 1™^/* ^ *" combtaed 

ity code directed to the merge controUer 46. That third °f object untoconddewticm 

priority code also comprise the absolute Z" value of which toinftp^ofa giva plane. In object cooidmates 

the processing element from one or the other of two ^ " *y expression: 

groups of four processing elem«>ts whichwill contrib- 45 ^tr+W4)+i(r +m+<KZ*mfi^-K (0 

ute to a given puel m the final miage. The compare- ^ r^, ^ r^, 

exchange tree of the merge control processor 46 in- where 

eludes compareHawhange blocks 138-150. as weU as x. Y and Z are olriect coordinates: and 
logic Wock 152 which generates an enable signal to the ^' 1,^,^ ^^^'^'aSTpTOC^g element of 
dday buffer 80 of the select«l processing group. Re- 50 "^tje 64 MOMsSng eten^id 
c«pt of the enable signal at that delay buffer 80 of the ^» C id n^prSeteSinS constants. 
S=^nil«rrtL'SoSlot^^^^ E^;e,io«(.)nutybfre.r«nged.sfonow.toyie.d: 
'"lSS^rLto£»n^^?;)%)an^ .«*««.0(Z,-,-^^-.,^-C«^ 
6(c), a merge scheme is employ«l which permits the expression (2) 64 different constant vahies KO 

formation of a combmed image from the mim-miages ^mnigh K63 may be determined, one for each process- 
produced by the mdividual processmg elements. The dement as foUowK 
combmed image is formed, preferably in raster scan 
order, on a pixel-by-pixel basis. Such an approach re- 60 

duces pipeline latency and permits real-time user inter- xo - iC - Aktfi* - Bk^ - O«o64 (j) 

action 10 with the combined image. Moreover, this id • if - Akxit — BliU - Oii|64 

reduction in pipeline latency is achieved with a rela- 
tively simplified hardware configuration over that set 

forth in the aforementioned paper. £5 „ 

In addition, in accordance with an important aspect 
ofthe present invention, the priority codes which deter- KO - k - AktM - Blf}6* - CmtiU 

mine which particular processing elem&t will contrib- 
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All of these values KO through K63 are computed by this b accomplished by means of a stored template 
tiie host 18 based upon the desired position of the slice which represents the projected area of a typical voxel in 
plane. two dimensions. The stored template must be varied 
The slice controller 50 of FIG. 7 includes three regis- depending upon the relative orientation of the object. 
tei8 200,202and204towhich values of A, Band Care 5 The template must also be varied depending upon the 
directed from the host 18 and stored. The outputs from scale fiictor by which each voxel is projected and also 
registers 200, 202 and 204 are directed to multipliers by the shape of the voxel, i.e., whether cubic, rectangu- 
206k 208 and 210, respectively. Also directed to the kr, parallellapiped, etc. In carrying out the foregoing 
multipliers 206, 208 and 210 are X, Y and Z addresses the host 18 determines the template for each orientation, 
fh>m the coordinate transform unit 48. The multipliers 10 scale factor and voxel shape employed. 
206, 208 and 210 thus compute values AX, BY and CZ, In accordance with the present invention, therefore it 
respectively. These values are summed at adders 212 is desirable to paint in the CRT display not only the 
and 214, thus producing the quantity AX+BY+CZ pixel which represents the center of a given voxel but 
which is directed to a plurality of comparators 216, 218 also neighboring pixeb which represent the area within 
and 220, etc.» there being one such comparator for each 15 the stored template for that voxel. In carrying out the 
of the processing elements. Also directed to the com- foregoing a biifTer is provided which stores the com- 
parators 216, 218 and 220 are the outputs of a plurality bined image (D', Z")i that buffer being responsive to a 
of registers 222, 224 and 226, there being one such regis- template memory. The coherent image (D", Z'") from 
ter for each such processing element. The comparator the buffer includes points of the combined image en- 
216, 218 and 220 determine whether a given voxel is in 20 larged by the area of the template, 
front of or behind the selected slice plane and each If the projections of the template of two different 
generates the appropriate slice plane control signals voxels fall on the sam pixel it is also necessary to posi- 
which are directed to the individual processing ele- tion the closer of the two templates to the viewer and to 
ments. ignore the other at that pixel. In accordance with the 
The slice plane controller of FIG. 7 has been de- 23 present invention this is accomplished by comparing the 
scribed in connection with the deletion from the com- depth values of the two templates. The foregoing will 
bined image of portions of the object which lie in front be better understood by referring to FIGS. 8(a)-8(c). 
of a given plane, i.e., the removal from the combined As shown in FIG. 8(a), current values of the corn- 
image of points having a predetermined depth function bined image, i.e., D' and Z" are directed in raster scan 
with respect to the plane of the output display. It should 30 order to a serpentine buffer 230 which is responsive to 
be understood, however, that this depth function need the stored template memory 232 to yield a coherent 
not define a planar surface. For example, it may be image consisting of digital values referred to as D" and 
desirable to delete from the combined image all points Z'". The digital values of the coherent image, D" and 
on one side of a given curved surface, rather than a Z"' are a function of combined image D' and Z" and are 
planar surface, having a predetermined depth function 33 delayed in time therefrom. During that time delay, each 
with respect to the plane of the output display. |nxel of the combined image, D', Z" is compared against 
„ ^ • TT neighboring pixels in, for example, a 3x3 pixel 
i>can Conversion Umt y^^^^ ^ ^^^^^^ ^^^^ ^j^^ combined image, 

As mentioned above, the output from the merge con- D', Z" are substituted by values for neighboring pixels 
trol unit 46 is a combined image produced from the 40 which have been found to be closer to the screen of the 
outputs of each of the individual processing elements, CRT 22. Thus, if the current pixel (D', Z") has a ndgh- 
formed preferably in raster scan order. The combined boring non-zero pixel which is closer to the vantage 
image comprises a Z" or absolute depth value and a D' point of the viewer than the current pixel and if the 
or density value for each pixel to be displayed. The template of that neighboring pixel overlaps the current 
combined image is durected to the video post-processing 45 pixel, then the coherent hnage (D", Z'") is set equal to 
unit 52 which generates an enhanced image from the the density value (D') and the depth value (Z") of that 
combined image. The video post-processhig unit 52, neighboring pixel llils depth value (Z") may be ad- 
also mentioned above, preferably mcludes a scan con- justed by adtiUng a distance of&et stored hi the corre- 
version unit, 54, a shadmg processor 56 and a shadow spondhig location of the template memory 232. This 
processor 58, each of which is desirable though not 50 formof data interpolation has been found to be particu- 
necessary to produce the final enhanced image. The larly useful in yielding coherent images which do not 
combined image from the merge control unit 46 is first indude holes and without undue time delay, 
directed to the scan conversion unit 54 whidi eliminates In accordance with m example of the present in ven- 
the problem of holes mentioned above to yield a coher- tion. the template to be employed and against which the 
ent image. The scan conversion unit 54 will be de- 55 combined image is compared is a 3X3 pixel template, 
scribed m detail m connection with FIGS. 8(a), 8(^) and Thus, the combined coherent unage is time delayed by 

the scan conversion device 52 of the present invention 

Each value of the coherent hnage Le. D* and Z" by two scan lines and three pixels. It will be appreciated 
represents the density and absolute depth of the two that each pixel of the combmed hnage may be compared 
dimensional unage of a particular voxel. In fiict, m ac- 60 against its eight adjacent pixels during this time delay, 
cordance with the preferred embodunent each value of The serpenthie buffer 230 which permits the compari- 
the combined image, i.e., D' and Z" represents the den- son of a given pixel of the combmed unage with its 
sity and absolute depth of the center of a given voxel. neighboring pixels wiU be described m connection with 
Thus each three dimensional voxel is projected m two FIG. 8(6) below. 

dimensions as a single point It is necessary, however, in 65 As shown in FIG. 8(b). the incoming pixels of the 
order to form a coherent image, to project each voxd m combmed image (D', Z") are supplied to the serpentine 
two dimensions as an area as opposed to a pomt In buffer 230 which includes a matrix of nine distance 
accordance an important aspect of present invention comparator elements 234-250. After a time dday of two 
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scan lines plus three pixels, the last of the distance com- 
parator element 250 yields an output of the coherent 
image (D'\ Z'"). Each pixel of the coherent image (P'\ 
Z'") will be identical to its corresponding pixel of the 
combined input image (D', Z") uxiless it is determined 
by the distance comparator elements 234-250 that val- 
ues of that pixel should be replaced by values of neigh- 
boring pixels. Each pixel of the coherent image may 
differ from the corresponding combined input pixel if it 



ing to the depth, Z'" of the coherent image, the smaller 
or lower the depth value from the vantage point of the 
viewer, for example, the higher the value of the inten* 
sity. The intensity values fh)m the depth shading pro- 
cessor 272 are directed to a multiplier 280. The depth 
values Z'" from the scan oonversion unit 52 are also 
directed to the gradient processor 270 which yields 
intensity values corresponding to the orientation of the 
object with respect to a simulated light source, the more 



is witUn the template of another pixel and that other 10 oblique the angle of orientation with respect to that 



pixel Is non-zero and closer to the viewer. 

Referring now to FIO. 8(c), one of the distance com- 
parator elements such as 234 will be described hi detail 
As shown in FIO. 8(c), the distance comparator element 
234 mcludes an adder 256 to which depth values Z" of 15 
the combined image are summed with the dq>th values 
H (I, J) of the appropriate value of the template. The 
summation, H', is the distance to the front &ce of the 
voxel (as opposed to the center) from the viewer. This 



light source the lower the intensity value. The intensity 
values from the gradient processor 270 are also directed 
to the multiplier 280^ the output of which is directed to 
an adder 282 which combines the product of the mten- 
sity values from the gradient processor 270 and the 
depth shadmg processor 272 with a pre>determined, 
variable value of simulated ambient light. The output 
from adder 282, along with the output from the density 
processor 268, are directed to the shading look-up tables 



value, H', b directed to a comparator 258. Also directed 20 for red, green, and blue components 284, 286, and 288, 



to the comparator 258 is a value of Z/at which repre- 
sents the depth from the viewer to the front face of Uie 
immediate neighboring voxel. The comparator 258 con- 
trols a first multiplexer 260 which passes either the 
value of H' or Zm, depending on which is closer to the 25 
vantage point of the viewer, the value being directed to. 
a pipeline register 262. The comparator 258 further 
controls a second multiplexer 264 to which is directed 
the density value of the combmed image, D', and the 
density of the inunediate neighboring voxel, D/jsr. Ei- 30 
ther the density vahte D' or Dm are passed by multi- 
plexer 264 to pipeline register 266 depending on 
whether the current voxel or the neighboring voxel are 
closer to the vantage point of the viewer. The output of 
pipelme registers 262 and 266, ZouT and Docm are 35 
directed to the neighboring distance comparator ele- 
ment 236 as shown in FIG. 8(6). 

Shading Processor 



respectively. The outputs from the red, green, and blue 
shadmg look-up tables 284, 286, and 288 are respec- 
tively directed to multiplexers 290, 292, and 294. Also 
directed to multiplexers 290, 292, and 294 are the values 
from the corresponding color look-up tables of the 
density processor 268. The output of multi-plcxcrs 290, 
292, and 294 yield the red, green, and blue signals from 
the shading processor 56 which are directed optionally 
to the shadow processor 58 or, alternatively, to the 
frame buffer 60. The shadow processer 56 is described 
in detail in connection with FIG. 10 below. 

The mathemadcai procedure utilized by the shading 
processor 56 is set forth and described in a paper enti- 
tled "Image Shading of Three-Dimensional Objects", 
Department of Radiology Technical Report MIPG85, 
by Dan Gordon and R.A. Reynolds, published in No- 
vember, 1983 which is incorporated herein by reference 
as ifset forth in full. 
Referring now to FIG. 9(6), details of the gradient 
The coherent nnage, D", Z"\ from the scan conver- 40 processor 270 will be described in further detail. The 
sion unit 52 is directed to the shading processor 56 as gradient processor 270 comprises a serpentine buffer 
shown in FIG. 3 and as described in detail in connection which includes a series of pixel delays 400, 402, 404, 
with FIG. 9. The shading processor 56 as shown in 406, 408, and 410 in series with a pair of scan line delays 
FIG. 9 is responsive to the scan conversion means 52 412 and 414 of a type well known to those skilled in the 
and varies the intensity of portions of the coherent 45 art The serpentine buffer 270 yields a series of outputs 



image produced by the scan conversion means with 
respect to the intensity of other portions of the coherent 
image to yield reahstic shading effects. The output of 
the shading processor means 56 comprises a parallel 
series of 8-bit color signals including a red signal (R), a 50 
green signal (G), and a blue signal (B). 

The shading processor 56 shown in FIG. 9 includes a 
density shading processor 268, a gradient processor 270, 
and a depth shadmg processor 272, D" values of the 
coherent image from the scan conversion unit 52 are 55 
directed to the doisity shadmg processor 268, whereas 
depth values Z'" from the scan conversion unit 52 are 
directed respectively to the gradient processor 270 and 
to the depth shading processor 272. 



Z'"(+ 1, + 1) through Z'"(- 1. - 1)» each of which are 
delayed from the others. These outputs are directed to 
the circuit of FIG. 9(c) which computes the surface 
gradient normal of the object under consideration on a 
pixel-by-pixel basis. 

As shown in FIG. 9(c), only five (5) of the nme (9) 
possible outputs of the serpentine buffer 270 have been 
employed. Output values from the serpentine buffer 270 
are directed to subtractors or difference circuits 412, 
414, 416, and 418 as shown. The difference circuits 412, 
414, 416, and 418 compute the difference between the 
depth value Z"' of one pixel with respect to a neighbor- 
ing pixel. Trace differences are required in oi^er to 
detemtine the change m depth of the surface of the 



The density shading processor includes a memory 60 object under consideration from one pixel to its neigh- 



comprising three look-up tables, a red, green, and blue 
look up table, 274, 276, and 278, respectively. The red, 
green, and blue look-up tables, 274^ 276, and 278, pro- 
duce a pseudo color encoding well known to those 
skilled in the art. 

Depth values Z'" from the scan conversion unit 52, as 
mentioned above, are directed to the depth shading 
processor 272 which yields intensity values correspohd- 



boring pixel. The differences computed by the differ- 
ence circuits 412, 414, 416, and 418 are directed to a 
plurahty of look-up tables 420, 421, 422, 423, 424, 425, 
426 and 427 which provide a weighted differential, i.e., 
65 the greater the slope or difference between one pixel 
and its neighbor, the smaller the weight Such a 
weighting function is set forth and described at length in 
the aforementioned paper of Gordon and Reynolds. 
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for comparison with Z(i) at comparator 620. X(i), Y(i) 
and Z(i) are the coordinates of the point under consider- 
ation with respect to the simulated light source. The 
distance, ZO)* is compared with the absolute depth, Z'", 
from the previous ftame at the comparator €20, If the 
distance^ Z(i), from the simulated ^ght source to the 
point under consideration equals the distance between 
the viewer and the pobt (Z*"), the point under consid- 
eration is illuminated and control signal 618 causes the 
multiplexer 612, 614 and 616 to pass unattenuated red, 
green and blue intensity values to the frame buffer 60. 
Alternatively, if the distance between the simulated 
light source and the point under consideration, Z(i), 
differs from the distance between the point and the 
viewer (Z'*% then the point is considered to lie within a 
shadow region and the attenuated red, green and blue 
intensity values from the look-up tables 600, 602 and 604 
are passed to the CRT 22. 

Object Access Unit 

Referring now to FIG. 11, the object access unit 32 
will be described. The object access unit 32 is coupled 
between the host bus 30 and the object access bus 34 
and acts as an interface between the host computer 18 
all look up tables are loaded by the host 18 and may be 25 and the various individual processing elements 36-42, 
varied interactively to produce the desired shading etc. The object access unit 32 includes a protection and 
efiTect control means 702 for protecting data stored in the 

. ^ various processing elements from modification. Data 

:inaaow rroccssor ^^^^ j^^^^ ^ directed to the process and con- 

Referring now to FIO. 10, the shadow processor 58 30 trol unit 702 via databus 700. Data from the protection 



The weighted differential from look-up Tables 420 and 
422 are directed to a first adder 428 and to a second 
adder 430. The outputs of look-up uble 424 and look-up 
table 426 are directed to a third adder 432 and a fourth 
adder 434. The combined outputs from the adder 428 
are directed to a look-up table 434 whereas the com- 
bined outputs from adder 432 are directed to a look-up 
table 436. Look-up tables 434 and 436 implement an 
inverse function in accordance with the teachings of the 
aforementioned paper of Gordon and Reynolds, the 
outputs being directed to multipliers 446 and 448. Also 
directed to multipliers 446 and 448 are the sununadons 
from adders 430 and 444^ respectively. The outputs 
from multipliers 446 and 448 are directed to a look-up 
table stored in memory 450 which yields the output 
signal from the gradient processor 270. The output 
signal from the gradient processor 270 is a scaled signal 
varying between 0 and 1 which modulates the intensity 
of the depth shading processor 272 in accordance with 
the gradient of the surface of the object under consider- 
ation when compared to the direction of the simulated 
light source. The table stored in look-up table 450 is 
loaded by the host computer 18 for each orientation of 
the object under consideration. It should be clear that 
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win be described in detail. For any given pixel, the 
shadow processor 58 outputs a red, green and blue 
intensity value or an attenuated red, green and blue 
intensity value depending upon whether or not the 
point under consideration is within the shadow of a 35 
simulated light source or not. Incoming red, green and 
blue intensity values from the shadow processor 56 are 
directed to attenuation look-up tables 600, 602 and 604 
via line 606. Red, green and blue intensity values are 



and control unit 702 is directed to the object access bus 
34 by a databus 704. The object access unit 32 further 
includes an object descriptor memory 706. Information 
concerning each independent object under consider- 
ation is directed to the object descriptor memory 706 
via lin 716 from the host bus 30. The object descriptor 
memory 706 contains the identification of the particular 
processing elements 36-42, etc. which will be used to 
compute the mini-image for a given object or portion of 



farther directed to a shadow frame buffer 608 via line 40 a given object as directed by the host. 



: 610. Unattenuated red, green and blue values are di- 
rectly output from the shading frame buffer 608 to mul- 
tiplexers 612, 614 and 616 which operate under the 
control of a control signal 618 from a comparator 620. 
Depending upon the state of the control signal 618 to 45 
the multiplexers 612, 614 and 616, either an unattenuate 
intensity signal from the shadow frame buffer 608 is 
output to the CRT display means 22 or, alternatively, 
the attenuated red, green and blue intensity signal from 



The object access unit 32 further includes an address 
computation means 708 which is coupled to the host bus 
30 by a first address bus 710 and which is further cou- 
pled to the object access bus 34 by a second address bus 
71X 

When the host 18 transmits data concerning a given 
object to be displayed across host bus 30, the data is 
directed to the protection and control means 702 and 
the address of that data is directed to the address com- 



the attenuation look-up tables 600, 602 and 604 are di- 50 putation means 708 via address bus 710. The address 



rected to the frame buffer 60. The unattenuated or the 
attenuated intensity signals are directed to the frame 
buffer 60 depending upon whether or not the point 
under consideration is fully illuminated from a simu- 
lated light source or whether that same point lies within 55 
a shadow region with respect to that light source. In 
making this determination, two video frame times are 
required. In a first video frame time period, red, green 
and blue intensity values from the shading processor 56 



computation means 708 is responsive to the object de- 
scriptor memory 706 and computes the corresponding 
processing element address and the corresponding pro- 
cessing element selection code which b transnutted via 
line 714 to object access bus 34. 

In short, the object access unit 32 permits the host to 
select and assign various processing elements to com- 
pute the mhii-image of a given object and records and 
detemunes the appropriate addresses at which the ob- 



are stored in the shadow fcame buffer 608 along with 60 ject data is stored m the object memories of the respec- 
♦k. A^^^u ^u:^^ .v.* tive processing elements. 

While a particular embodunent of the present inven- 
tion has been shown and described, it will be appreci- 
ated that other modifications may be made withhi the 
spirit and scope of the present invention. These modifi- 
cations are to be included within the scope of the ap- 
pended claims. 
What is claimed is: 



the depth of the object for that pixel, Z'", which is 
received from the scan conversion unit 54. In the next 
video frame time period, the distance Z(i) from the 
simulated light source to the point under consideration 
is computed by the invene transform unit 622. Addi- 65 
tionally, X(i) and YQ) are computed by the inverse 
transform unit 622 which is used as an index into the 
shadow frame buffer 608 in order to select the point Z'" 
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L A real-time three-dimensional display device com- 
prising: 

(a) a plurality of individual processing elements each 
having: 

(i) an associated object memory for storing a first 
digital representation of a portion of a three-di- 
mensional objet to be displayed; 

(ii) an image memory for storing a second digital 
representation of a mini-image of the visible sur- 
face of that portion of the object as viewed from 
a selected direction with respect to a predeter- 
mined plane including the relative deptii of diat 
surfece at each pixel of the mini-iinage; 

CilQ means for selecting said direction including 
means for mapping said first digital representa- 
tion into said second digital representation; and 

(b) means for generating priority codes based upon 
the relative significance of a given pixel of the 
mini-image produced by a given processing ele- 
ment as compared to the pixels of mini-images 20 
produced by other processing elements for a given 
pixel in the combinedimage produced by all of said 
processing elements; and 

(c) merge control means responsive to said priority 
codes for generating the combined image of said 25 
object including^ at each pixel, the depth of the 
visible surface of the object from said predeter- 
mined plane said combined image being generated 
on a pixel-by-pixel basis from said mini-images, 
each pixel of said combined image including only a 30 
selected pixel of said mini-images which is visible 
for a given orientation of said object, the pixel 
being selected in accordance with said priority 
codes. 

2. The display device of claim 1 further comprising: 

(d) a video post-processing unit for generating an 
enhanced image from said combined image. 

3. The display device of claim 2 further comprising: 

(e) a frame buffer for storing said enhanced image. 

4. The display device of claim 3 further comprising: 
(0 a display means, responsive to said frame buffer, 

for displaying said enhanced image. 

5. The display device of claim 2 wherein said video 
post-processing unit further comprises: 

a scan conversion means for generating a coherent 45 
image from said combined image. 

6. The display device of claim 2 wherein said video 
post-processing unit further comprises: 

a shading processor means for varying the intensity of 
portions of said combined image with respect to 
other portions of said combined image depending 
upon the relative distance and orientation of such 
portions from a simulated light source. 

7. The display device of claim 6 wherein said shading 
processor means controls the intensity of portions of 55 
said combined image relative to the intensity of other 
portions of said combined image. 

8. The display device of claim 2 wherein said video 
post-processing unit further comprises: 

a shadow processor for adding shadow effects caused 60 
by a simulated light source to said combined image. 

9. The display device of claim 2 wherem said video 
post processing unit further comprises: 

0) a scan conversion means for generating a coherent 
image from said combined image; 

01) a shading processor means responsive to said scan 
conversion means for varying die intensity of por- 
tions of said coherent image with respect to the 
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intensity of other pordons of said coherent imag^» 
and 

^ a shadow processor responsive to said shading 
processor for adding shadow effects caused by a 
snnulated light source to said coherent image to 
produce said enhanced image. 

10. The display device of claim 9 further comprisuig: 
(e) a frame buffer responsive to said video post- 
processor unit for storing the enhanced image pro- 
duced thereby. 

11. The display devke of claim 10 fUrther compris- 
ing: 

(0 a display means responsive to said frame buffer for 
displaying the enhanced image of said object. 

12. The display device of claim 1 wherein said means 
for generating a priority code further comprising: 

logic means for generating a low priority code for 
pixels in the mini-image of a given processing ele- 
ment whenever the object memory for that pro- 
cessing element contains no digital representations 
of said three-dimensional object. 

13. The display device of claim 1 further comprising: 
means for controlling said mapping means whereby 

the mapping of said first to said second digital rep- 
resentation may be selectively controlled. 

14. The display device of claim 13 wherein said 
means for controlling selectively permits the mapping 
of points from said first to said second digital represen- 
tation having a predetermined set of densities. 

15. The display device of claim 13 wherein said 
means for controlling selectively permits the mapping 
of points from said first to said second digital represen- 
tation having a predetermined depth function with re- 
spect to said predetermined plane. 

16. The display device of claim 13 wherein said first 
digital representation includes a time dimension and 
wherein said means for controlling selectively permits 
the mapping of points from said first to said second 
digital representation having selected time sequences. 

17. A real-time three-dimensional display device 
comprising: 

(a) a plurality of individual processing elements each 
having an image memory for storing a mini-image 
generated by that processing element including the 
relative depth of each pixel of such mini-image; 

(b) means associated with said processing elements 
for generating said mini-images; and 

(c) a merge control means for generating a combined 
image from each of said mini-images by selection 
based upon the relative significance of a given pixel 
of the mini-image produced by a given processing 
dement as compared to the pixels of mini-images 
produced by other processing elements for a given 
for a given pixel m the combined image produced 
by all of said processing elements on a pixel-by- 
pixel basis said combmed image including at each 
pixel the depth from a predetermined plane. 

18. The display device of claim 17 farther compris- 
ing: 

(d) a video post-processing unit for generadng an 
enhanced image from said combined image. 

19. The display device of ckum 18 further compris- 
ing: 

(e) a frame buffer for storing said enhanced image. 

20. The display device of claun 19 further compris- 
ing: 

(0 a di^lay means, responsive to said frame buffer, 
for displaying said enhanced image. 
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21. The display device of claim 18 wherein said video 
post-processing unit further comrpises: 

a scan conversion means for generating a coherent 
image from said combined image. 

22. The display device of claim 18 wherein said video 5 
post-processing unit Aurther comprises: 

a shading processor means for varying the intensity of 
portions of said combined image with respect to 
other portions of said combined image depending 
upon the relative distance and orientation of such 10 
portions from a simulated Ught source. 

23. The display device of claim 22 wherein said shad- 
ing processor meaiis controls the intensity of portions of 
said combined image relative to the intensity of other 
portions of said combined image. 15 

24. The display device of claim 18 wherein said video 
post-processing unit further comprises: 

a shadow processor for adding shadow effects caused 
by a simulated light source to said combined image. 

25. The display device of claim 18 wherein said video 20 
post processing unit further comprises: 

(i) a scan conversion means for generating a coherent 
image from said combined image; 

(ii) a shading processor means responsive to said scan 
conversion means for varying the intensity of por- 25 
tions of said coherent image with respect to the 
intensity of other portions of said coherent unage; 
and 

(iii) a shadow processor responsive to said shading 
processor for adding shadow, effects caused by a 30 
simulated light source to said coherent image to 
produce said enhanced image. 

26. The display device of clahn 2S ihrther compris- 
ing: 

(e) a frame buffer responsive to said video post- 35 
processes unit for storing the enhanced image pro- 
duced thereby. 

27. The display device of claim 26 further compris- 
- ing: 

(f) a display means responsive to said frame buffer for 40 
displaying said enhanced image. 

28. Ilie display device of claim 17 further compris- 
ing: 

(d) means for controlling said mini-image generation 
means whereby the generation of each of said mini- 45 
images is selectively controlled. 

29. The display device of claim 28 wherehi said 
means for controlling selectively permits the generation 
of mini-hnages having a predetermined set of densities. 

30. The display means of claim 28 wherein said means 50 
for controlling selectively permits the generation of 
mini-images having a predetermined depth function 
with respect to a predetermined plane. 

31. The display device of claim 28 wherein said 
means for controlling selectively permits the generation 55 
of mini-images in a selected time sequence. 

32. A real tune» interactive, medical image display 
system for displaying data representing a three-dimen- 
sional anatomical object, sdd data having been col- 
lected by at least one medical image acquisition device 60 
comprising: 

(a) a host computer for receiving and storing said 
data; 

(b) an image display processor coupled to said host 
computer, said image display processor having: 65 
CO a pluraUty of individual processing elements 

each having an image memory for storing a mini- 
unage of the visible surface of a portion of said 



32 

object as viewed from a selected direction with 
respect to a predetermined plane including the 
relative depth of that surfoce; 
00 means associated with said processing elements 
for selecting said direction and for generating 
said mini-hnages; and 
(iii) a merge control means for generating a com- 
bined image from each of said mini-images by 
selection based upon the relative significance of a 
given pixel of the mini-unage produced by a 
given processing element as compared to the 
pixels of mini-images produced by other process- 
ing elements for a given pixel in the combined 
image produced by all of said processing ele- 
ments on a pixel-by-pixel basis, each pixel of said 
combined image including the depth of the visi- 
ble surface of the object from said predetermmed 
plane; and 

(c) display means, coupled to said image display pro- 
cessor, for displaying said combined unage. 

33. The system of claim 32 wherem said image dis- 
play processor further comprises: 

Civ) an object access unit coiq)led between said host 
computer and each of said mdividual processing 
elements for assigning subsets of said data to se- 
lected ones of said processing elements. 

34. The system of claim 32 wherein said image dis- 
play processor further comprises: 

Qv) a coordinate transform means, coupled to said 
host computer, for generating mapping signals to 
each of said individual processing elements, said 
mapping signals varying the min-image produced 
by each of said processing elements depending 
upon the direction selected with respect to said 
predetermined plane. 

35. The system of claim 32 wherein each such indi- 
vidual processing element of said image display proces- 
sor further comprises: 

(iv) a coordinate transfonn means, coupled to said 
host computer, for generating mapping signals to 
such processing element, said mapping signals 
varying the mini-image produced by that process- 
ing element depending upon the portion of the 
object or objects whose mini-unage b computed by 
that processing element 

36. The system of claim 32 wherein said unage dis- 
play processor further comprises: 

(iv) control means coupled to said host computer and 
to said mdividual processmg elements for selec- 
tively controlling the generation of said mini- 
images. 

37. The system of claim 32 wherein said image dis- 
play processor further comprises: 

(iv) a video post processmg unit for generatmg an 
enhanced image from said combined image. 

38. The system of daun 37 wherem said video post 
processing unit f\irther comprises: 

a scan conversion means for generating a coherent 
image from said combined image. 

39. The system of claim 37 wherein said video post 
processing unit further comprises: 

a shading processor for varying the intensity of por- 
tions of said combined image with respect to the 
intensity of other portions of said combined hnage 
depending upon the relative distance and orienta- 
tion of such portions with respect to a simulated 
light source. 
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40. The system of claim 37 wherein said video post 
processing unit farther comprises: 

a shadow processor for adding shadow effects to said 
combined image. 

41. The system of claim 32 wherein each of said indi- 5 
vidual processing elements of said image display proces- 
sor further comprises: 

an object memory for storing a first digital represen- 
tation of portions of said objects; 

an image memory for storing a second digital rcpre- 10 
sentation of the mini*image of that portion of said 
object; and 

a means for mi^ping said first to said second digital 

representations. 
4X The system of dahn 41 wherein each of said indi- 15 
vidual processing elements of said image display proces- 
sor fbrtber comprises: 
means coupled between said object memory and said 
image memory for controlling the mapping of said 
first to said second digital representation. 20 

43. The system of claim 42 wherein said first digital 
representation includes a time dimension and wherein 
said means for controlling selectively permits the map- 
ping of points from said first to said second digital rep- 
resentation having selected time sequences. 25 

44. The system of claim 42 wherein said means for 
controlling the mapping means permits the mapping of 
points from said first to said second digital representa- 
tion having a predetermined set of densities. 

45. The system of claim 42 wherdn said means for 30 
controlling the mapping means permits the mapping of 
points from said first to said second digital representa- 
tion having a predetermined depth fimction with re- 
spect to a predetermined plane. 

4^. The system of claim 41 wherein said first and said 35 
second digital representations each include a density 
dimension and wherein each such individual processing 
element further mcludes: 
a volume feature counter for accumulating the num- 
ber of points stored in the mini-image of that pro- 40 
cessing element having a non-zero density dimen- 
sion, said host computer being responsive to the 
volume feature counters of each of said processing 
elements whereby the volume of said anatomical 
object may be automatically computed. 45 

47. Am image display device comprising: 

(a) a plurality of individual processing elements, each 
for computmg a mini-image of a portion of an ob- 
ject and each having an image memory for storing 
that mini-image including the relative depth 50 
thereof; and 

(b) a merge control means for generating a combined 
image from each of said mini-images by selection 
based upon the relative significance of a given pixel 
of the mini-image produced by a given processing 55 
element as compared to the pixels of mini-images 
produced by other processing elements for a give 
pixel in the combined image produced by all of said 
processing elements on a pixel-by-pixel basis said 
combined image including the depth at each pixel 60 
from a predetermined plane. 

48. A real-time three-dimensional image display de- 
vice comprising: 

(a) a plurality of mdividual processing elements* each 
for producing a mini-image of a portion of a three- 65 
dimensional object; 

(b) merge control means for generating a combined 
image of said object on a pixd-by-pixel basis from 



each of said mini-images, each pixel of said com- 
bined image having a density value and a depth 
value relative to a predetermined plane, said com- 
bined image as generated by said merge control 
means having insufficient data to yield a coherent 
unage at certain orientations of said object as said 
combmed image is projected on a display means; 
and 

(c) scan conversion means responsive to said com- 
bmed image for generating a coherent image de- 
spite said insufficient data said scan conversion 
means having: 

(j) a seipentme buffer for delaying each given pixel 
of said combined image; and 

0i) a template memory, said serpentine buffer being 
responsive thereto, for storing neighboring pix- 
els of said delayed pixel, the density value of said 
given pixel being substituted by the density value 
of a neighboring pixel whenever said neighbor- 
ing pixel has a non-zero density value and a rela- 
tive depth less than the relative depth of said 
given pixel. 

49. The image display device of claim 48 wherein said 
template memory stores the density value and depth 
value of eight neighboring pixels. 

50. A real-time, three-dimensional image display de- 
vice comprising: 

(a) a means for producing an image on a pixel-by- 
pixel basis, each pixel of said image having a den- 
sity value and a depth value relative to a predeter- 
mined plane; 

(b) a shading processor responsive to said image for 
adding shadhig effects to said iamge, said shading 
processor having: 

(0 a gradient shading processor responsive to said 
depth values, for computing the surface normal 
of each pixel of said image on a pixel-by-pixel 
basis; and at least one of 

(ii) a density shading processor responsive to said 
density values; and 

(iii) a depth shading processor responsive to said 
depth values; and wherein said shading proces- 
sor further includes a means responsive to out- 
puts of said gradient shading processor and at 
least one of said density, and depth shading pro- 
cessors for generating a final image on a pixel- 
by-pixel basis, the mtensity of said final image 
being varied in accordane with the depth value, 
or the density value and the surfee normal for 
each pixel. 

51. A real-tune, interactive display system compris- 
ing: 

(a) a host computer; 

(b) an image c^play processor coupled to said host 
computer, said image display processor having: 

(i) a plurality of individual processing elements 
each having an image memory for storing a mini- 
image generated by that processing element in- 
cluding the relative depth; 

(ii) means associated with each of said processing 
elements, for generating said mini-images; and 

(iiO. merge control means for generating a com- 
bined iamge from each of said mini-images by 
selection based upon the rdtive significance of a 
given pixel of the mini-hnage produced by a 
given processing element as compared to the. 
pixels of mini-images produced by other process- 
ing elements for a given pixel m the combined 
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image produced by all of said processing ele- 
ments on a pixel-by-pixel basis each pixel of said 
combined image including the depth at that pixel 
from a predetermined plane; and 
(c) display means, coupled to said image display pro- 
cessor, for displaying said combined image. 
S2. The system of daim 51 wherein said image dis- 
play processor further comprises: 
an access unit coupled between said host computer 
and each of said individual processing elements for 
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controlling the mini-images generated by each 
processing element 

53. The system of either of claims 32 or 51 wherein 
said merge control means generates said combined 
image in raster scan order. 

54. The display device of claim 17 wherein said 
merge control means generates said combined image in 
raster scan order. 
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